Right: Normalized attention scores processed by two different normalization methods. Table 1: Performance of searched architectures using different NAS algorithms in DARTS [ 7 ] space on CIFAR-10 [ 5 ]. The inference latency was measured on a machine with GeForce RTX 3090 GPU. The batch size was set to 1. Encode(ms) Infer(ms) Total(ms) NAR-Former 2.4784 17.4864 19.9648 NAR-Former V2 2.3722 5.2276 7.5998 may be somewhat different. Due to the softmax, Eq. ( 5) focuses almost all attention on the current The Eq. ( 2) restricts attention to connected nodes by introducing the adjacency matrix.

As more deep learning models are being applied in real-world applications, there is a growing need for modeling and learning the representations of neural networks themselves. An effective representation can be used to predict target attributes of networks without the need for actual training and deployment procedures, facilitating efficient network design and deployment. Recently, inspired by the success of Transformer, some Transformer-based representation learning frameworks have been proposed and achieved promising performance in handling cell-structured models. However, graph neural network (GNN) based approaches still dominate the field of learning representation for the entire network. In this paper, we revisit the Transformer and compare it with GNN to analyze their different architectural characteristics. We then propose a modified Transformer-based universal neural network representation learning model NAR-Former V2.

Right: Normalized attention scores processed by two different normalization methods. Table 1: Performance of searched architectures using different NAS algorithms in DARTS [ 7 ] space on CIFAR-10 [ 5 ]. The inference latency was measured on a machine with GeForce RTX 3090 GPU. The batch size was set to 1. Encode(ms) Infer(ms) Total(ms) NAR-Former 2.4784 17.4864 19.9648 NAR-Former V2 2.3722 5.2276 7.5998 may be somewhat different. Due to the softmax, Eq. ( 5) focuses almost all attention on the current The Eq. ( 2) restricts attention to connected nodes by introducing the adjacency matrix.

As more deep learning models are being applied in real-world applications, there is a growing need for modeling and learning the representations of neural networks themselves. An effective representation can be used to predict target attributes of networks without the need for actual training and deployment procedures, facilitating efficient network design and deployment. Recently, inspired by the success of Transformer, some Transformer-based representation learning frameworks have been proposed and achieved promising performance in handling cell-structured models. However, graph neural network (GNN) based approaches still dominate the field of learning representation for the entire network. In this paper, we revisit the Transformer and compare it with GNN to analyze their different architectural characteristics. We then propose a modified Transformer-based universal neural network representation learning model NAR-Former V2.